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oo Abstract 
O 

A clinical study was performed comparing the efficacy of multimedia learning modules with 
traditional textbooks for the first few topics of a calculus based introductory electricity and mag- 
netism course. Students were randomly assigned to three different groups experiencing different 
presentations of the material; one group received the multimedia learning module presentations and 
the other two received the presentations via written text. All students were then tested on their 
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learning immediately following the presentations as well as two weeks later. The students receiving 
the multimedia learning modules performed significantly better than the students experiencing the 
text-based presentations on both tests. 
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I. INTRODUCTION 



A. The instructional dilemma in introductory physics 

How to best spend lecture time to promote student learning in introductory courses is a 
topic that we often debate with ourselves and with our colleagues. Recent research suggests 
that actively engaging students in learning activities results in better conceptual understand- 
ing 1 compared to passively listening to lectures. There is also evidence that students are 
better prepared to learn difficult concepts from lectures 2 or lecture demonstrations 3 when 
they are aware of what they don't know and don't understand. The common thread running 
through this diverse set of studies is that there is more and deeper learning when students 
come to class with some basic exposure to the content so that they are prepared to learn 
from the experiences in lecture. Reading the textbook prior to coming to class is one way of 
providing the necessary basic exposure to content, but getting students to read the textbook 
is a difficult goal to achieve. 

Data from a survey administered to students at the University of Illinois at Urbana- 
Champaign (UIUC) over three consecutive years, shown in Table [l| indicate that both life- 
science majors and engineering majors seldom, if ever, open the textbook prior to coming 
to class. About 70% of both groups admit they "rarely" or "never" read the textbook; 69% 
of both groups claim that the textbook was either "useless" or "not very useful" in helping 
them understand the course material. What makes these findings surprising is that at UIUC 
we attempt to enforce textbook reading prior to class time by administering Just-in-Time 
Teaching (JiTT) 4 "preflight" questions crafted from the readings. These data indicate that 
students answer the preflight questions based on intuition, or merely to fulfill a requirement, 
rather than from an informed perspective. Based on the comparability of UIUC students 
to students nationwide, and on the frequent complaints voiced by physics professors across 
the country that students come to class unprepared, we believe that students' "textbook 
irrelevance" attitude is quite prevalent. 

These circumstances create a dilemma. On the one hand, research suggests that we 
should actively engage our students during class and that instructors' expertise is best used 
to coach students in learning the subtleties of physics concepts and problem solving; but this 
requires that students come to class with some exposure to basic content and with some idea 
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of what they do and do not understand — something that we know students are not doing. 
On the other hand, recognizing that students come to class unprepared to participate in 
meaningful activities suggests that we should spend class time covering basic content from 
the textbook; but this would leave little time for coaching students in developing conceptual 
understanding and problem solving skills. 

B. Multimedia learning 

One possible solution to this dilemma is to use multimedia presentations to help students 
learn basic content prior to coming to class. There are three reasons why multimedia pre- 
sentations might prove effective in this context. First, today's generation of students has 
grown up in a multimedia age, and they not only know how to use multimedia better than 
the current generation of physics professors, but they also like using multimedia. Second, 
presenting basic physics content via asynchronous, web-based multimedia presentations can 
be monitored and enforced better than policing textbook reading. Multimedia modules can 
be designed with embedded assessments, and a modest amount of course credit can be as- 
signed to viewing the materials (so long as some minimum time is spent on them) and to 
performance on the embedded assessments. 5 

The third reason why multimedia presentations of basic content might provide a viable 
solution to the textbook-irrelevance problem is that a considerable body of research exists 
on how to design multimedia materials to improve learning. This is in stark contrast to 
the situation with textbooks, where enhancements continue to be added (e.g., addition of 
color; boxes containing problem solving tips, reflection questions, and questions to quiz 
oneself) with no evidence on whether they add, or perhaps detract, from student learning. 6 
Principles for designing multimedia materials draw from both cognitive science and empirical 
studies. For example, work on cognitive load over the last twenty years by Sweller and his 
collaborators 7 indicates that if the student is using valuable working memory resources in 
order to simply coordinate images and words, s/he has less capacity for actual learning. 8 
The expanding research base in the field of multimedia learning has led to the following 
conclusions: 9 a) people have separate audio and visual channels, b) these channels have 
limited capacity, and c) learning involves the active selection, organization and integration 
of the information presented via the auditory and visual channels. 10 
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Several design principles have emerged from multimedia learning research findings. For 
example, students learn better (%) from words and pictures than from words alone, (it) 
when words and pictures are presented simultaneously rather than successively, (Hi) when 
extraneous words, pictures and sounds are excluded, and (iv) from animation and narration 
than from animation and on-screen text. Although most of these are intuitive, one principle 
that is perhaps somewhat counterintuitive is that students learn better from animation 
and narration than from animation, narration and on-screen text. Overall, this body of 
research suggests that in designing multimedia learning materials the goal should be to stay 
on-message with respect to the learning goals and to use different input channels (visual, 
auditory) to provide complementary but necessary information to help the learner build 
meaning and understanding. 

C. Question to be addressed 

The current situation can be summarized as follows: Students don't read the textbook 
as much as we would like and textbook reading has proven difficult to enforce. On the other 
hand, viewing and answering embedded questions in multimedia presentations delivered 
via the web could be more closely monitored and awarded credit, encouraging students 
to conduct the requisite pre-lecture preparation. There is, however, no research at the 
university level examining whether or not students learn basic physics content better from 
multi-media presentations compared to reading the textbook. Before any major effort is 
devoted to designing and implementing multimedia presentations in large enrollment courses, 
we need to know if they are at least as effective as reading a textbook in terms of imparting 
basic content knowledge. 

Thus, an obvious question to ask is: How well do textbook presentations fare compared to 
multimedia presentations in teaching introductory physics content? The study reported here 
is an attempt to answer this question. More specifically we compared three different modes 
of presenting the content of the first two weeks of a typical calculus-based introductory 
electricity and magnetism course: 1) Paper presentations based on a popular introductory 
textbook, 2) Web-based multimedia presentations designed according to principles from 
multimedia learning, and 3) Paper presentations consisting of the script from the web-based 
presentation in item 2. We were interested in comparing these three modes of presentation 
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in terms of assessments of learning both immediately following the presentations, as well as 
after an extended period of time as a measure of retention. In addition, we asked students 
two attitudinal questions regarding their use of textbooks in the course, and their preference 
for the materials used in this study compared to their current textbook. 

II. DESCRIPTION OF THE STUDY 

A. Choice of topics 

We specifically chose the material for our study from the first two weeks of an intro- 
ductory, calculus-based electricity & magnetism (E&M) course. Topics included Coulomb's 
Law, superposition, electric fields from point charges and continuous distributions, electric 
flux, and Gauss' Law. Students find E&M concepts very abstract and difficult, making them 
ideal for investigating which type of presentation mode works best for helping students learn 
basic abstract content. 

B. Description of presentations 

1. Web-based, multimedia learning modules 

Four pilot multimedia learning modules (MLMs), were developed: 1) Coulomb's Law, 2) 
Electric Fields, 3) Electric Flux, and 4) Gauss' Law. The MLMs drew heavily on research 
related to multimedia learning summarized above. Each MLM covered one lecture's worth of 
course content, and was divided into approximately 10 scenes. Each scene was implemented 
as a Flash 11 movie containing dynamic animations synchronized with an audio narration 
that was controlled by the student (pause, play, rewind, and position). Embedded formative 
assessments were included in two or three of the scenes for each module, which consist of 
questions that must be answered correctly before the student can move to the next scene. 
The average narration time for an MLM was 12 minutes; students typically took about 17 
minutes to complete a single MLM including the embedded assessment. 12 
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2. Scripts from multimedia learning modules 



One of the two paper-presentation modes consisted of scripts of the MLM narrations, 
(heretofore referred to as MLM-script) together with accompanying static pictures from the 
ML Ms' animations. Each MLM-script was about 7 pages long and contained 7 figures. 13 

3. Textbook-based materials 

The second paper-presentation mode consisted of verbatim material from a traditional 
textbook (heretofore referred to as the Textbook group) subject to the constraint that the 
same worked out examples covered in the MLMs had to also be covered in the textbook-based 
materials. The reason for this constraint is that the cognitive research literature suggests 
that students often focus on worked out examples to learn how concepts are instantiated 14 , 
and so we wanted to keep the worked out examples in all presentations equivalent. We 
selected materials from the Tipler and Mosca textbook 15 , largely because the material and 
worked-out examples in this textbook paralleled those in the MLMs. We deleted a total of 
six worked examples from the four textbook units that were not covered in the MLMs. We 
also added one example from the Halliday, Resnick and Walker 16 textbook to match the one 
example in MLM-script that had no counterpart in the Tipler & Mosca materials. Each 
edited textbook unit was about 7 pages long and contained 8 figures and 2 examples. 

C. Study procedures 

Volunteers were solicited from students taking the first semester calculus-based mechan- 
ics course (Physics 211) at UIUC and offered compensation in exchange for participation in 
the study. Students were paid $15 for each of three 90-minute sessions, and a $20 study- 
completion bonus. Students were randomly assigned to three groups, each receiving a dif- 
ferent treatment (MLM, MLM-script, or Textbook). 

The four units were delivered within two 90-minute sessions distributed over two days in 
the same week. Each 90-minute session consisted of two "lessons," each lasting about 45 
minutes. Each lesson began with a presentation (the MLM group sat at computers to receive 
the multimedia presentation; the other two groups received paper booklets containing either 
the MLM-script, or Textbook presentation) and was followed by a post-lesson assessment. 
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Two weeks later students returned for a retention test, which lasted about 1 hour. 

D. Assessments 

1. Post-lesson assessments 

The Post-Lesson Assessments consisted of multiple-choice and free-response questions; to 
streamline this article and discussion of results, we will only present results from the multiple 
choice questions. The multiple-choice questions were largely conceptual; some were taken 
from the Conceptual Survey or Electricity and Magnetism 17 and some were constructed by 
us. The 43 multiple choice questions on the Post-Lesson Assessments may be obtained from 
the authors. 

2. Retention Test 

The Retention Test administered about two weeks following the lessons had 37 questions 
(both multiple choice and free-response), 33 of which were either identical or very similar 
(e.g., surface feature changes, change in order of answers) to questions on the Post-Lesson 
Assessments, and 32 of which were multiple choice. Only results from the multiple choice 
questions will be presented here. The 32 multiple choice questions from the Retention Test 
may be obtained from the authors. 

3. Survey questions 

At the end of the third "retention" testing session, the following two survey questions 
were administered to all three groups: 

1. Compare the presentation of the material in this study, with that of your current 
textbook 

(a) Strongly prefer current textbook 

(b) Somewhat prefer current textbook 

(c) About the same 
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(d) Somewhat prefer these study materials 

(e) Strongly prefer these study materials 

2. How often do you now read your physics textbook? 

(a) Two or more times per week 

(b) About once per week 

(c) About 1 to 3 times per month 

(d) Just to cram for exams 

(e) Never 

III. RESULTS 

A total of 63 students who were enrolled in the calculus based mechanics course volun- 
teered for the study and were randomly assigned to one of the three groups. The results 
reported here are based on the 45 students who completed the entire study (16 MLM, 13 
MLM-script, 16 Textbook). The average Physics 211 exam scores (based on combining scores 
on the three course hour exams) for each of the three groups, along with its uncertainty, 
are shown in the second column of Table [II} The spread is about what one would antici- 
pate statistically, with a slight advantage to the MLM-script group. We will provide raw 
results, and results that use the covariance to adjust for differences between the groups as 
determined by these mechanics exam scores. 

First we will address how the three groups compare in terms of initial learning immedi- 
ately following each lesson by analyzing the students' performance on the 43 multiple-choice 
questions of the four Post-Lesson Assessments. Then we will compare the three groups on 
the retention test administered two weeks following the lessons by analyzing the students' 
performance on the of 32 multiple choice questions from the Retention Test. 

A. Comparison of multimedia learning modules vs. textbook 

The combined results from the Post-Lesson Assessments and associated error are given 
in column 3 of Table [TTJ The MLM group had an average score of 75% compared with the 
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Textbook group's score of 63%. The standard deviation of the Post-Lesson Assessments scores 
is 13.5%, which means students in the MLM group scored nearly a full standard deviation 
higher on the exams than their peers who used the textbook (effect size = 11.5/13.5 = 
0.85). Since the Textbook group's mechanics exam scores were slightly lower (1.5%) than 
those of the MLM group we can use the correlation between the mechanics exam scores and 
the Post-Lesson Assessment scores to estimate an adjusted score for the Textbook group of 
65% as shown in column 4. Including this adjustment the MLM group still scored nearly 
10 percentage points higher than the textbook group giving an effect size of about 0.7. 
Performing an analysis of covariance based on the mechanics exam scores and the Post- 
Lesson Assessment scores, we obtain the result p < 0.01 18 . Hence students using the MLMs 
demonstrated a large and statistically significant advantage in performance on the Post- 
Lesson Assessment items compared to students using the textbook. We now address the 
question of whether or not this advantage perseveres by looking at the Retention Test. 

The Retention Test was given approximately two weeks after the lessons, and the raw 
results are shown in column 4 of Table II Once again the MLM group performed about 13% 
higher than the Textbook group, with an average score of 70% compared with the Textbook 
group's raw average of 57%, which gets adjusted to 58% based on their mechanics hour exam 
performance. The adjusted effect size is 0.7 with p < 0.01. Once again we have a large, 
statistically significant result showing that students using the MLMs not only learn more 
immediately following the lessons, but also retain that information better than the students 
using the textbook presentation two weeks following exposure. 

Figure [T] illustrates the significance of this effect. Each dot represents a student; on the 
horizontal axis is their average mechanics hour exam score, and on the vertical axis is their 
average combined score including both the Post-Lesson Assessments and the Retention Test 
(labeled Study Scores on the graph). The line in the figure represents a linear fit to all 
of the data. Notice that 75% of the MLM group scores are above this line, while 75% 
of the Textbook group scores are below it. Hence, independent of the students' ability 
(based on their mechanics score), more learning and better retention resulted from the 
MLM presentations versus the Textbook presentation. 
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B. Comparison of MLM-Scripts, with MLM and Textbook Presentations 



According to multimedia learning theory 19 , the large differential in performance in favor 
of the MLMs over the Textbook is likely attributable to two factors: 1) MLMs dispense 
information via dual channels (visual and auditory), and 2) MLMs create less memory load 
compared to the textbook. The motivation for including the MLM-script group in our study 
was to test the multimedia learning theory further by comparing its effectiveness to both the 
MLM and the Textbook. The MLM-script presentation and the Textbook presentation differed 
in one important way according to multimedia learning theory: Multimedia learning theory 
states that presentation layout should minimize cognitive load by excluding extraneous words 
and pictures (the "coherence principle" 20 ); in short, a presentation should stay strictly "on- 
message" with little else to distract the learner from that message. This principle, coupled 
with the "dual channel" principle, would predict that: 1) the MLM-script group should 
outperform the Textbook group (not only because the "enhancements" in textbooks serve 
to distract students from staying "on-message," but also because the MLM-scripts were 
"spartan" presentations, with a tight story-line, augmented by the few crucial figures needed 
to illustrate the ideas in the text), but 2) The MLM-script group should perform lower than 
the MLM group (because, although both were "on-message," the MLM-scripts lacked the 
dual channel processing available in the MLMs). 

Our findings were in accord with these predictions, as shown in Table [IT] and Fig. [2] The 
MLM-script group's scores are above those of the Textbook group, but below those of the 



MLM group. Table [Hl| shows the effect size S and the p-value for all possible comparisons 
of the three groups for both the Post-Lesson Assessments and the Retention Test. The p 
values were computing using ANCOVA with the physics 211 exam scores as the covariant. 

C. Results of two survey questions 



Table IV and Fig. [3] summarize the results for the two survey questions. We report 



the results of the first question for each group separately, and the second question for all 
students together (this second question did not depend on the treatment received; it only 
asked students to report on their use of the current textbook in their course). 

As can be seen from the results of the second question summarized in Fig. [3j students 
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in this study reported similar usage of their textbooks as those in Table |T} Only 16% of all 
students who finished the study reported that they read their textbooks once per week or 
more and 50% claim to never have used the textbook the entire semester! 

The results from the first question support the statement made earlier, namely that 
today's generation of students prefer to learn from multimedia materials compared to text- 
books. The survey results for question 1 indicate that the MLM group showed a greater 
preference for the study materials encountered during the intervention over their traditional 
textbook as compared to the Textbook and the MLM-script groups (p < 0.02, Fisher ex- 
act test). 21 (For this comparison the two text-based presentations were combined). The 
response pattern between the Textbook and MLM-script groups did not significantly differ 
from each other (p = 0.4 Fisher exact test). 

IV. GENERAL DISCUSSION 

Our study has shown that learning of basic physics content by the current generation of 
undergraduates from a typical modern introductory textbook fared poorly when compared 
to learning from multimedia modules that were designed based on principles derived from 
research into multimedia learning. The effect sizes obtained in this study are consistent 
with those from numerous other multi-media learning studies. For example, Mayer and 
collaborators have performed a series of 39 experiments designed to compare the retention 
performance of groups using presentations that either were, or were not prepared according 
to the seven design principles they developed and they found positive effects in 33 of them 22 
with the average effect size being 0.8. We conclude that multimedia learning modules rep- 
resent a viable, and more easily enforceable alternative means of pre-lecture preparation for 
introductory physics students. 
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Tables 



TABLE I: Results of two questions from a survey administered in 2003, 2004 and 2005 to students 
completing introductory physics courses at UIUC 







Algebra-based physics 


Calc-based physics 






for life-sciences 


for engineers 


How often do you 


Always 


13 (2%) 


32 (5%) 


read the text before 


Regularly 


70 (10%) 


55 (9%) 


attending class? 


Occasionally 


112 (17%) 


116 (18%) 




Rarely 


151 (23%) 


133 (21%) 




Never 


321 (48%) 


297 (47%) 


Overall, how impor- 


Essential 


9 (1%) 


30 (5%) 


tant was the text in 


Very Useful 


31 (5%) 


49 (8%) 


helping you under- 


Useful 


160 (24%) 


114 (18%) 


stand the course ma- 


Not Very Useful 


254 (38%) 


173 (28%) 


terial? 


Useless 


208 (31%) 


256 (41%) 
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TABLE II: Performance of the three groups on hour exams, Post-Lesson Assessments, and the 
Retention Test. 



Mechanics course Post-Lesson Assessments Retention Test (# of ques- 

exam scores (# of questions = 43) tions = 32) 

Raw Adjusted Raw Adjusted 

MLM{n = lQ) 78.5 ±3.1 74.7 ±2.6 74.7 69.7 ± 4.0 69.8 

Textbook (n = 16) 76.9 ±2.3 63.2 ±3.1 65.2 56.8 ±4.1 58.1 

MLM-script (n = 13) 81.4 ±3.3 71.7 ±3.0 69.7 65.1 ± 3.7 63.0 
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TABLE III: Effect size (S) and significance (p) for comparisons of the performance of the three 
groups on both Post-Lesson Assessments and the Retention Test. 



Post-Lesson Assessments Retention Test 





MLM 


MLM-script 


MLM 


MLM-script 


Textbook 


S = 0.74, p < 0.01 


S = 0.36 p = 0.06 


S = 0.67, p < 0.01 


S = 0.31, p = 0.16 


MLM-script 


S = 0.46, p = 0.03 




S = 0.45, p = 0.04 
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TABLE IV: Results of survey questions. 



Survey Question 1: Compare the presentation of the material in this study, with that of 
your current textbook 

a) Strongly prefer current textbook 

b) Somewhat prefer current textbook 

c) About the same 

d) Somewhat prefer these study materials 



e) Strongly prefer 


these ; 


study materials 




Response 


MLM 


MLM-script 


Textbook 


a), or b), or c) 


3 


8 


8 


d), or e) 


13 


5 


8 
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Figure Captions 

Figure 1: A Scatter Plot of a student's average mechanics exam score vs his/her combined 
Post-Learning and Retention Scores for both MLM and Textbook groups. 

Figure 2: Average adjusted scores on the combined Post-Learning Assessments and the 
Retention Test for all three groups (MLM, MLM-script, and Textbook) 

Figure 3: Survey results for frequency that student uses textbook in current physics 
course. Since exams occur about once per month, we have grouped choices c) 1-3 / month 
and d) cram for exams into one category labeled every month. 
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Figures 



FIG. 1: A Scatter Plot of a student's average mechanics exam score vs his/her combined Post- 
Learning and Retention Scores for both MLM and Textbook groups. 
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FIG. 2: Average adjusted scores on the combined Post-Learning Assessments and the Retention 
Test for all three groups (MLM, MLM-script, and Textbook). 
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FIG. 3: Survey results for frequency that student uses textbook in current physics course. Since 
exams occur about once per month, we have grouped choices c) 1-3 / month and d) cram for exams 
into one category labeled every month. 
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